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(57) Abstract 

An apparatus for combining a number of video signals into a single composited video signal is described. Inverse motion compensating 
devices (60) are used to transfoim motion compensated transfoxm domain video signals to conespondudg transform domain video signals. 
A compositing mechanism (62) receives the transform domain video signals from the inverse motion compensating devices and performs 
transform domain video compositing operations directly on the transform domain video signals. The compositing operations result in a 
single composited signal incorporating each of the transfoim domain video signals. The single composited signal may be encoded for motion 
compensation (64) to generate a single motion compensated transform domain video signal suitable for transmission. After transmission, 
inverse transfoim domain operations and inverse motion compensation operations (52) may be performed to reconstruct the composited 
signal. The composited signal may then be projected on a video display device (56). 
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METHOD AND APPARATUS FOR COMPOSITING 
COMPRESSED VIDEO DATA 

Brief Description of the Invention 

The present invention relates generally to the 
processing of video signals. More particularly, the 
present invention relates to a method and apparatus 
5 wherein multiple video signals, each in a compressed 
format, are combined in the compressed format to render a 
single composited scene. 

Background of the Invention 

10 Video compositing is the process of combining 

multiple video signals, originating from diffierent 
sources, to produce a single composited scene. Video 
compositing is commonly performed in multipoint video 
conferencing, video editing, and multimedia applications. 

15 Typical video compositing operations include 

processing of individual video objects by general 
geometrical transformation or filtering, or combining 
multiple video objects by opaque or semi-transparent 
overlap. In generWlV^Vl'dfeo objects are' semi-transparent, 

2 0 arbitrarily^shaped^;' and arbitrariay-positioried • ; Dif ferent 

compositing operations have different complexity. Most 
applications,!|only implement a subset of the compositing 
operatipns. , ;i / - 

.For; network video applications, video compositing can 

, 25 be impl,emgjnte<d at different locations, including the video 
source " sites", intermediate -r.^nodes within or outside 

* networkg^^l; an??; users^'^' display, sites. Although - video' 

cbmposiiEirig can be implementedJin the uncompressed- spatial 
domain, -w!her^l operations are. performed pixel by pixel; 

3 0 most va,d.ep dat:a transmitted through networks, or stored ifi 
. video servers, is represented in some compressed format^: 
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Therefore, it would be desirable to perform video 
compositing operations on data which is still in a 
compressed format. 

Many video applications utilize data compression. 
5 More particularly, many video applications utilize 
transform code cbmpressed^ cicinain formats (referred to 
herein as "transform domain" formats) , which include the 
Discrete Cosine Transform (DCT) format, the interframe 
predictive code format, such as the Motion Compensation 

10 (MC) algorithm, which may be used in conjunction with' the 
DCT format, and hybrid compressed formats. The DCT format 
is used ,in the -compression standard for still' images JPEG 
^(Standard Draft, JPEG-.9-R7, Feb.: 1991). The combination 
of , Motion. Compensation and rDiscrete Cosine Trahsform 

15 compression, algorithm ; (MC/DCT) r is used ih a - number^ of 
standards Including:, the. compres,sion Jstahdard for * motion 
pictures (MPEG. ~ , s^a^d^^rd. -^^^ Committee 
Draft, MPEG 90/176. Rev. 2, Dec , 199G) , the standard for 
video conferencing . (CCITT Recommendation H.261,' Video 

20 Codec for Audiovisual Services at px64 kbits/s), and some 
High, Definition Teleyisipn-i proposals . - 

Figure 1 is. a block diagram depicting . the processing 
associated with the DCT :f o,rmat . ) : At-the\ encoder^ 20 ; the 
DCT' format is established ,^ by. DCT , f ormulator 22, 

25 Specifically, the video, data, is segmentedi' into blocks ^ of 
data, or pixel blocks, ..which, can be- represented by ari N x 
N matrix, which^ is referred ; to ,as; V A" * The matrix A of 
video input . da,ta can^ .be transfprmed to . the *DCT domain 
through the following .operation:. : )_ ^ i ■ ' 



30 



A^ = CAC?"^ 



1 



where. C ( i, j ) ^. ^ , ,=^\ m j t ; . *^ : r 



1 ' 



- 2 - 
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for i=0 and 



10 



M 



Zcos-iLi2jil)i 



tThe .cT -term refers to the; transposition of the C inatirix, 
svi.ch ;^thLat the columns . of <C become irdws in and the rows 
. in. C • become, :.)the..:.caliumns in C^; *^ The eleinentfs' (i, j } 
represent the spectrum of the^'^ original image block;** A, at 
.. different -spatial -f reqitehcies / 'Elements Vith larger' index 
values represent! the fii^Wef-^i^eqlidhcy compbnents'. * ' ^- 
15 . Typically, :th^' DCT'- coefficients A'ji,j)' are thin 

. quantized at quantizer' 24. ■ Triat" is;"' the ^ individual 
. coefficients of the -resOitarit^' N*"''x"'^N^ matrix ar4 each 
transformed to a finite number of ^ tiits. " Usually/ the 
■n number of bits - is (Na^^ 'furibtion of the rate ' and quality 
2 0 recjuirement of the -video^ service J ' ? .^.^ 

' 'Typically, i the quantiz'ed' va-liies are further encodeci 
...by. the variable ilengtli^^coder 2 6; which of ten includes a 
run-l,ength.' (coding 'function and an entrdpy ' coding function . 
: ..The runr length coding -function use's a single bit length to 
25 , represent a aong ruil^of 'zero values . Th^ entropy "coding 
function assigns longeir 'bit patterns to' sampled with a 
higher probability of occurrence, and shorter bit patterns 
to samples with a lower probability of occurrence. 
Consequently, the overall average data amount can be 
30 reduced. The variable length coded data is' then 'conveyed 
to transmission channel 28. 

As depicted in Figure 1, inverse operations are 
performed at decoder 3 0 to return to an uncompressed 
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domainl In particular, inverse variable- i^hgth coding is 
performed by inverse variable length coder 32, an inverse 
' quantization step is performed by inverse quantizer 34, 
and an inverse DGT operation'' is perforined by inverse DCT 
5 fonnulator 36." This proc^sising^ results ^in -the original 
uhcoinpressed' video data, ' : ^ - ' - ' 

' Usually,^ the DCT-'coinpfressed ' v signal 'has iiiuch 

less data 'than 'the - original' spatially uncompressed video 
signial. Therefore/ ' '^ivleh" multiple DCT-cbmpfessed Video 
10 signals, compositing directly in the DCT-compressed domain ' 
- is a potentially^ e"fficient approach/ That is, -given DCT- 
compressed data input video' streams, it would be desirable 
to directly perform' compositing bperatiohs ■ 6ri^4:he ' bCT- 
^ * compressed data , ' wi thbiit cohversidn to the ■ un^ 
15^ s;patial domain: / - ' However, - 'tliere^ arfe cpmf>lications ■ 
' associated with transform domain ' dompositing ' o^erati 

In many of" the cbAip'riessiori standards ' previously 
-mentioned, the DCT ■ algorithm ' i^ accompanied by the 
interframe motion ^compen&atidn * ^ (MC) ; ' algbrithm which 
20 exploits the temporal reduri^^ in the video sequence . 

•r fAs previously Stated, this datarfornlat is referred to as 
MC/DGTv : The MC : algorithm^ incl?udes two types of data: 
motion veGtor data a-nd^ error ■ prediction data , . 

* The MCi algorithm searches: over- a* ^fixed-size area, 
25. • called the motion area, and identifies -the ' optimal^ 
reference .block in - a previous, video - frame which may be 
iused to construct an image for ' a » block ' Within a present 
• video frame; > - Figure 2A depicts^^ the location : of a 
reference block 4 0 from a previous video' frame 'which may 
30 be 'Used to ' construct - ^a curteht block 42 in a present " 
frame. The motion vector defines the difference between 
'-the position of the reference block '40' and the position of 
. -the, ^ current block 42v The ipredietlbh error defines the 
^ difference in* image ^content between the reference block 
35. and the current block. ^ ^. - M ; r 

- The MC encodingoprocess may bis 'Characterized by the 
following reguation: p * . ~ - . .^w 
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.where e(.t,x^y):, is the prediction* error at .time ,t.,. and 
coordinates (x./yj)., P.J.s the . .cu'rr^ant image-block, P^^^ ds/.the 
5 reconstructiBd JLinage> . and d-.-is the motion jvector . 

Figure 2B illustrates that . the. reference block . 4 0. may 
forin a portion ,o.f four yid^eo -blocks , namely, B^, B^, B3, and 
^ . B^ . ^ , . Th^ ^problems ^associated , with ^converting such a 
, - . ref ^risnce iplpck to a, purrent ^ block, will ; be discussed 
10, \ ^below... . . •• • . . ■ i'. - V' ' - ' J ; ; • : • • 

.. . -^..Figure 3 11 lust-rate problem agspciated; with the MC 
; _ alg9r;i.thin .when a foreground'. object ,37 partially overlaps 
a, bac:tcgrpund,. object. 38 , Assume that , the,,foreground.. object 
^ is . totally ppaqye.^ the block ^boundary positions for 

15 ^ t:hese . two obj^pts . match. The, strippe,d: , area 39 ;Wdll be 
, referred to ^ as : tliL^^.^,.'»c^ii^ec^ effected . area" , -because, part 
of the ;notion^ area is, .<Jip^cJ;iy^. reiplaced. by the foreground 
^ ^ object ,.3^7 ... It- is necess.ary. t;o ^checki every cur*rent b-loc):^ 
. in -tfyB f .diarectly/- .affected: areqtr . to aee if its reference 
20 iDlocJc Is replaqed by the foregroundi: object 37. . If 'sa?,: we 
need to .. recalculate':. the MC data, . bincluding : a 'new motion 
Xectpr and new prediotion errors;: ^of the current block. 
The remaining- parti ojLxtside rthe ditrectly. affected- arfea in 
, .the backgrounds object.^: .38 - is * called the ^ '^iiidirectly 
2 5j , (Effected area" 41.,- because its ; reference clocks could be 
_ located, in the . ".directly . affected area" 39 > whose MC data 
may ,nejed itp .i?e recalculated. . . In this case> the motion 
vector , is unchanged, . ..but p the prediction earrors may . have 
. miTioi;,.di£f erencQS-w ^. .^c" v- . tift; : - :i. < 

3 f -i Ariother. example pirpblem with compositing MC/DCT d^ta* 

r^eJLateP: to ^scaling,. Suppose, it ds desirable to scale down 
, ^n.. image biy ao.ratio ..pf ; -2i to -1 on ..each /side; . As shown^ in 
| Figurp, .4, fppp, n^ighbaring current i blocks 42A, i42B, 42C, 
420 will . be .scaled , down :to become a new ;blockt 43;;i v^JEach of 
35 the four current blocks has its own> iref erence block 40A/^ 
..,^ 40B, ;40C., ,4PD f Eom the .^previous .image >.frame .: r^> As .-shown in 
Figure 4, the reference blocks do not rnecessarily form the 
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same 2' by 2 area as' the current blc^ic^/ ' Thvis ; ' 4f ter the 
scaling operation: the reference bl^ocks' • a^W mapped to 

■ • different current blocks. Therefore, recalculatibn' of the 

■ MC data o* the new -down-scaled image 43 is' necessary ■ 

5.. - . . AS described, given -MC. data', including- motion vector 

■ -and prediction errors,. for the'input video sTghklsV 'one is 
■not able to caactilate new Me data for the c6inp6slted\;ideo 
• signal, airfectly-: ^ it is ne'ce'ssary to reconstruct the video 

- signals-.back in- the uncompressed spatial do:nain or tihe DCT 
10 compressed domain, -per fork compositing ^ ^^^^^^^^ and' 
calculate-, the nfeW MC datta- * or the composited video 
- Signals. AS mentioned earlier,: €he DCT comferes^ed domain 
has a lower data ' rate and thus potentially offers a more 
^ . "^^f ^""^ -approach tha^. the spatial ^ ^dbmWih.' ' ^ However, 
15 ■ techniques for converting vicibb. signals' -from the MC/bcT 
format to- the DCT format' aire hot ^m'o^^ < - ■ 

^ - ■ .oner skilled in - the^ art- wiil-recdgni^i that ^ the 
recalculat.on of.th^-MC dat^;..e^p6cially=^earchi^^ f6r the 
20 - ' P-vi6us - frW, Requires ' more 

20: computations than th^ -- process = to reconstruct ' the^ video 
sxgnal from^ the MC dat^ .n ThusV' it would ' b'e " desitable to 
r- provide a »-thod. and ^apparatus- for reducing 'com^^^^ 
associated, with the refaalcula^ibn of MC data. " ' 

in view Of thes^ problems; prior arf MC^DCT data 
processing apparatus rely upon full Conversion to the - 
uncompressed , spatial domain ^for compositing 6^>krations. 
one such .apparatus is portrayed in " Pi^re 51 • Figure 5 
depacts ; a compositing apparatus- ; in accdrcia^ndfe with' ' the 
prior art. / The figur^ represents that a first ihput video 
signal is conveyed to a first MC/DCT encoder 50 and a ' 
second input video signal is 'conveyed to a second MC/DCT 
encoder 50 of the same- constfubtibn, ^s will be described 
below. - Each MC/DCT encoder 50 produces an" ertbr signal 
in the DCT compressed- format, arid a motion' vectof signal' 
Both signals are then conveyed over a trarisinission chWnr^ei ' 



25 



30 



35- 

• 2 8B; 
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. . After transmission,., the signals , are received by 
, . MC/DCT. decoders ,52, which, jWill be described below^ The 
, . MC/pCT, decpders. 52 transform the signals , into the spatial 
domain.^ This? is. necessar.y,,- an accordance rwith the prior 
5, ,art , . in ..prdLer M to coirbine/ : ^^Qr composiite, the signals; 
Compositing, is- Der formed by a spatial dpina in compositing 
unit 54 which perfprins cpmp.psiting operations on a pixel 
by pixel basis. The .putput. .of the; spatial domain 
^ compositing "unit. 54 is..^, -..single. ; composited: signal, 
10 incorporating the two sets . of image data,.; , ... , >. 

^.Th^ resultant composited video -.signal- -can be 
. displayed pr\ a video display device at . the spatial ^domain 
compositing unit 54. If the. compositing unit^ 54 is at 
some intermediate , node, the compogited video, signal must 
15., be . reT$:ransformed ^ t-o the . cpmpresse^d formats by MC/DCT 
encoder 50. The MC/.DCT;^^pcQcJer . 5 the jcomposited 

signal and , re-t;rapsf ^rms, . it^^. intp the • MC/DCT,; domain,, 
thereby , generating, error,., andj,.j^ The^ 
pCT compressed error, .ajnd. motion, ^vector: .signal&uare then 
2 0 transmitted^ through transmi-s^iioni; ,chann^^ 28C,. ; Aft erf 
transmission J to, the ultimate usert^^destination .i the. terror 
and motion vector . signals,-,rMare transformed r into a 
reconstructed J videp signal j , by MC/D.GTt decoder -52. The 
reconstructed vidpp . sigpal; is . then )pro jected^ on a video 
25. display device .56.. ^ , ... i , . ?. - • 

Figure. , 6 ; depicts, .a ; video display- . device 56C, 
projecting, a .number of images 57;, typically .seen in a 
video^ conferencing application. , vThe . individual ' images 
57A, _57B,, 57C^,. 57D;, 575.,;,in. sum,, f.prm, a single composited 
30^^ image.. .... ...... : j .^ u-..r ■ - * 

..... . . Figur^ 7 d<spict§..an .MC/DCT^ en^ 

. with the^prior .a^t , ..The .input, video, signal is conveyed to 
.a , motion .vectpr, calculator, r 53..^^ r The; imption . vector 
calculatpr 53. receivesj the previous frameuid^ta (from frame 
35 memory .58,.,. The. mpt.ion. .vector, .calculator: : 53 compares ^the 
previous frame data with the present frame data to 
identify a reference block (as block 40 in Figure 2A) . 

- 7 - 
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The motion vector indicates ' the ■po^^itiohal ' difference 
•between the current = image block and the reference image 
. -block. ., This motion vector -f orms one output of ' the MC/DCT 
encoder 50 . , , .. 

5 The motion vector calculator 53 also generates the 

reference image block which is subtracted 'froni the input 
. video frame to ; produce an error signal.' ^ The error signal 
corresponds to the difference in' content between the 
reference block^fand the^' current- block. ' ■ ' 

10 ' ;The error signal ' is conviByed to ' DCT f orr^iulator ' 22B 

and quantizer ! 2 4'B to convert it to the DCT ■ddmain '. The 

• result i.of this processing is an error signal at 'the' output 
> of the. MC/DCT' encoder 50..; . ; < . i • , .: 

■ ■. ^ ' The .error signal . is 'valso'- conveyed -aidrig a ' fe'edback 
15- path, as ■reflected in Fi-gure -Tv. i'lW-th^ feedback path, an 
; 'inverse quantizer 34'B ande an' inverse - DGT fdrkulatdr -356 

• are used ito ; convert the error 'signa^i -^bacK- to the spatial 
domains The spatial, domain 'error signal is then- added to 
the previous reference block to form recbhstructed video 

2 0, data. The reconstructed fvadeb dat^ is ■ conveyed to the 
frame memory 58 and will; be: used; as the "previous frame" 
>, in'the niext, frame cycle:. : :"-i.U:- ;• - ? 

It .should be recognized, that the outputs Of the 
MC/DCT encoders 50 inay be further encoded with a variable 
25 length coder 26, For the sake of simplicity Variable 
length coders 26 will be omitted from future figures, 
although it will be appreciated -th^t. they may be , used in 
accordance- witii the invention. ..• i -'■ ■■' • f ;. 

/Figure 8 depicts the operation of a MC/DCT decoder 52 
30; in .accordance with -the -prior "art. : The inputs to the ' 
MC/DCT decoder 52 are the error signal and' the motion 
. vector .data. The error signal is processed by an' inverse 
quantizer 34C and an inverse DCT f ormulatdr ' 36C to produce 
an error signal within the spatial" domain. The ' spatial 
35 domain.: error signal is -added with reference block data 
from the frame memory 59. -.The combination of the 
reference data and the spatial domain error signal result 

.- 8 - 
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in the reconstructed video* which is :the o.utput. of the 
^ MC/DCT' decoder. . 52 The * reconstructed video, is also 

conveyed I to the >frainet. memory • for use in the next frame 
cycle. • 

^ Summary : of^ the; Invention s . fl T . ^ - - i . - • ; . 

• ,An .apparatus for ^combining a number^ of video, signals , 
, into-.^a . single composited . v.ideo .signal is, t. described . 
Inverse motion compensating devices are used to; transform 
10 ^motion; iCoiripensated transform . domain video signals to 
J. corresponding transform domain . video z.signals.- » . A 
compo&iting , mechanism f receives , the. transf orm domain* .video 
signals from the inverse motion compensating devices, and 
^. performs transform , domain . video ^.compositing operations 
15, directly^ .on .the ^transfprm ;.d:omain; video, signals,. ^ . . The 
compositing^ operations- result in. a> single composited 
signal^. dncorporating . eachi of the transform domain video, 
signals;. The sfingle compQS it ed signal may be encoded fpi? 
-motion compensation, to • -generate j a. single: motions 
20^ . compensated transforn domains video signal suitable, fpr^ 
I ^.transmission.' . After: transmissionv inve^rse > transform 
domain and inverse motion compensation j opera t-ions :nay be 
performed to reconstruct . ;the composited signal . The 
composited - sig^naL m^y then be projected^ on a- vidieo display 
2S, . devi,c.e, s^^, ... • il.-.: r.;. i , j- • = - <: 

. Brief Description of.?the .Drav/inas .. . ; ^ 

For a better understanding of the nature and- objects 
of ,the , inv.eipi.ti,oi5, ireferexice. should j.be' made- : to the 
30 , following .d:etaiied description taken in conjunction with- 
. the accompanying ^rawings/ in which: ;^<: i j\ ^ ' .>^; 
, ; V. FIGURE i.a.-j; depicts ; the f prior art signal f~ processing 
.associated; ,with\r.thei. Discrete Cosine Transform; - :., ir* , 
, . FIGURE .;2Ar idepicts V the; ^relationship-: between- a 

3 5 . :refer:enee ^ block and^ ; a currient . block for h a motion 
compensated. yid,eo signal. « : ^ .< i j> . 
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FIGURE 2B depicts- the prot,iei;^ -' o/ ^aeming" the 

- pos^Ltion Of a reference blbck' When the ^reference block 

• forms a 'portion of a number'^ of ' frame bldciks / ' ' ' 

■^^ -FIGURE '.;3:r. ^iiiastrdtes- a com^o^itihg problem 
5. 'experienced • in- the ^MC/DCT^dbxnafn WhWparir' bf th"^ motion 

■ area of a background iinagd'^is replaced ^^by ^'a^^ W 
^ » image.' : u. ■ - 'f i ', . f.-- ^ ■ w -.^ 

FIGURE M ^illustratek a- compositW ' prbblexn 
experienced .in-, the • .vfc/DCi- - •^dbmain Vhen scaling ' is 
10 performed, -j' . > ; ' ; i 

- - FIGURE. 5= depicts- a prib^^ ^ art" appa^atilir for 

• ' °o»P««iting MC/DCT^ domain video data in the" spatial 

domain. *' iJi^v-:^ <: \' ii- • -r. 

- FIGURE 6 depicts- a vic^^o aiilay- 'device with a number 
of .images ^composited into a ^^^ie dispi^y iinage ' 

FIGURE 7 depicts ^ pi^i^o^ 'a^t M^^/idT domain ' eni:oder 
as employed in tR^ appa'r£€uk 'bf ' figure 5 ''"^ ^ ' ' 

■ ^ ^FI^URE 8 depibt^ '^a pri^r-'art Jic/DcifJ domain decl^dar 
•'••as .employed i'n' tiiie'^iippB±d^yjl£-f^^--p^^r^-'.^ ■ • . . ' 

20 FIGURE 9 depicts ^n afoaratus^ iii accordance with the ' 

• xnventicn,' for ddmpbsitii^g' motion '^coriensated' transform 
; domain vildeo signa'Is. ^ .^v : j, , ; 

FIGURE 10 depict-^ ; a ' ^transf c^rm ■dimain motion 
compensating decoder ^ which ;is uskd in the' apparatus of 
25. Figure 9. ^ ^- : ■ ■ , :^ -^.' - ^ :^\> . \ ■ ^ ^ 

■FIGURE 11 ciepicts.the^transiatic^ri' operation' Which is 
. performed in accordance with thW invention 
. FK.URE . 12- . d^pict^ a trarisfbrin- <aomiin "^.otioh 

compensating .encoder which iV 'iibed ir^ thW appaa^atus ^ of 
Figure 9* ■ ^= • r;- ■ r:, ■ ^ ^ -r; , 
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FIGURE 13 depicts- an ap^ar^atus ^of ex^^^^^^ the 
functions associated with the^pres'ent irtveritibn " 

FIGURE :i4 illustrated - tfie donv^^^^^^^^ 
associated with . the transf brm' " domain ' filtering of the 
35 ■ invention. '-j.'-' ^ . > { c ■ 

• Li^ke reference numerals r^fW to' Wres^^^^ 
throughout the several views of the drawings. 

- io - 
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Detailed Description of the. Invention 

^ Figure 9 depict;s an, ^apparatus which may be, used., in 

accordance .with the invention,,, to- perform compositing 
operations , on , a motion compensated transform domain video 
5 signal . .^^^he^^,inyentio.n . willrp disclosed by using the 
MC/DCT compressed domain by, way of .example,. However, it 
will be appreciated that other transfoinn formats in 
conjunction motion, > compensation .;:arei : equally 

applicable to the invention. . Epr the sake of simplicity, 
10 the invention will be disclosed by describing ^ the 
compositing of two video signals. It is understood that 
a plurality, of yideo^ signals, mayj be processed • in 
accordance with the invention. 

Figure 9 depicts first and seeped input video- signals 
15 respectively being; conveyed, to., first and .second MC/DCT 
encoders 50B, which ^are ^bf the type previously . described. 
Each MC/DCT encoder 5QB .generates a DCT .compressed, error 
and ^ itiotion yectojc signal^.., whichr,, is suitable for 
transmission over transmission., c^^ / In the .prior 

2 0 art, the error ^and motion vector : signals would ^ be re-* 
transformed to the spatial^ dqir|ai^i> .at ..this, jiuncture.; .. prior 
to compositing. However, in accordance, with the .present 
invention, this st^p,,i^ eliminated,, ,L w- ' ' 

Instead of a^ copve.rsion .from the.: MC/DCT domain--to the 
2 5 spatial domain^ the present invention utilizes a transform 
domain motion comfiensatlr^g (MC) , decoder . 60 to:^ perform an 
inverse MC operation. Which : is . sufficients, to. . avoid an 
inverse DCT opearatipn; an^ otherwise-enable compositing in 
the DCT domain. The ..tr^ps^form domain MG decoder 60 of. the 
30 invention is disclosed in relation to Figure 10. . i i 

The transform domain MC decoder*, of- thew. invention 
includes an inverse, ^^gu.^^^ 34D. .of - the-.type previously 

described, a, reference bloc calculator ,61, }and l a frame 
mempry 63 . The. inv.erse quantizer -34D processes:^ the error 
35 signal. The reference block calculator 61. receives 
present mptipn . ve^ctpr daj;a and reconstructed DCT data from 
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a previous frame. ' The referenda ' blodk"^ '6^1^^ 61 
calculates' DCT ' data of the reference block. '''' 
■■ ' . MathematiGally /.the operation of the trahsforin do^^ 
i MC^decoder 60 maf be. .characterized by thk Mc' transform 
5.-. domain, decoder ^ equation^ t^hich • is ^def ihe'd as' 'fdH6ws : " 

DCT (P^^^ (t, X, y) ) =DCT(e (t , x,y) ) +DCT (P^^^ (t-i , ic-d^, y-d ) 



10 



Prec is the recpnstructed.image a^t time (t) and coordinates 
(x,y). Th^ fiinctioh 4 is the prediition error, which is 
described in relation to a time (t) and coordinates in an 
(x,y) grid. The value d in the last term of the equation 
is the motion vector. Note that all terms are in the DCT 
domain. Farther, note that .ths>, inverse DCT cpara^ is 
15 not performed.. ■ . .. . .... 

. Thus,, the transfprm domain, ,MC, decoder 60 performs an 
, operation .wherein, the DCT. .coefficients... of :the prediction 
error are. added to the - DCT .coefficients . of- a reference 
image block from a former frame, as provided by the 
20 reference block, Gialcylator , 61. , . 

■ ^''^.iP''^^^ M±th p^is,:^ar>prq^ch..,is that the motion 

. yector of each block is not, restricted to the fixed block 
boundary used in,, the RCT algorithm., That, is, the fixed ' 
block.boundary,.pf tne DCT algorithm cannot always be used 
25 to identify the. location of the reference block. . 

This. problem was illustrated in . reference to' Figure 
2B. , .^lote in, Figui-e 2B that reference block (B«) ' 40 
comprises four segments: B^, , B,,r^B^„: and' B,,, which' are 
respectively positioned within four blocks: B^, B^, B, , and 
30 B^. In other words, the reference block (B-) is a non-grid 
aligned object within the DCT grid. Thus, alignment of 
DCT coefficients is required. 

In accordance with the invention, a method is 
provided wherein the DCT coefficients of a new arbitrary- 
35 position image block are derived directly from the DOT 
coefficients of four original neighboring blocks. 
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Returning , to . Figure .2B^ the DCT coefficients of 
reference bloc,kj B', have ^contributions from four old 
neighboring, blocks .namely , the. r.upper-left. corner (8^3) of 
. , blpckt B^j, .the lower-left .^corner— (Bj^) of biockt Bg, the 
5 loyer-righ% earner: (B^^) pf.^blpck B^-, .and. the i upper-right 



corner 



(B^j) of block 



^4- 



Mathematically, this is 



characterized as : 



DCT{Bn =DPT 



0 »B. 
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+DCT 



0 

0 0 



' [00 



^42 0 



10 



.15. 



20 



25 



.Note, all sub-^b^locks*, B13, B24, B31; b42, may have a 
smaller size than the DCT blocks of which they ' form a 
: portion : Bl / B2 / g3 / \!B4 ;^ ^T6 conye^r^ sub-biock to a 

full dimensioned- DCT bl6ck, each sub-block is augmented to" 
..a full N by> N block -by -stuff ang zei"0 values, as will' be 
; described below;^^ , >;i - x ' -.l o v 'r :-i ' ^ . . . 

Figure 11 shows the mWemeht 'b£' Wub-block' Bli of old' 
jblock HI to its position: in reference -^b Note 
.the - upper-left ^ cbrneir-* o^^* the o iDlock BT' is 'extracted ,^ 
. supplementied ^ by zeros >'i 5ahd -moved to' the ^ loWer-'right 
..corner*. This process- ^is i necessary since the'' upper- I'eft^ 
sub-block^ of the ' old- block Bl will appear as' tihe lower- 
bright sub-tblodk of the referehce^block ' 40 (B'^ in Figure 
2B) . '.This operatiori can be -performed through the 
following, calculation: .,^^ - ' ^ * 



whet-e 



0 0 



0 H 



30 



-.13 - 
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Q 0 



10 



I,-and.I„,are idenfcity.matrides with size h" x-i-h' and^ w x w 
, respectively,; where ^h .and V ar^ number of rows and columns 
. extracted vfroxn,>bl.ock .:.B,. - .Art identity matrix is a featrix 
wxth "one" vaaues. across a diagonal of the matrix fr6m the 
- upper., .left corner to . the • lower-: right corner; with^ all 
other values .being zero-. : . < . ,. - , , . , , , 

As-, shown., in,, Figure 11; rmultij^lying E' with^ a 'pre- 
.^atrix .H, extracts the f irst h rows^ and .rahslate^ them to 
the. bottom.; multiplying. B, 'with a^po^t-mat.ix extracts 
the first w, columns and translates them to thfe right 
... Using the fact ..that . DCT . is distr^ibutive to matrix 

. inultxplxcation, ,we can . compute .the -DGT of b,, direitiy from 
,;..the,DCT Of :B,, i.e.,/,., i - . " ... . . 



15 



DC7 



0 B. 



13 



£)CT(S,) Ccr(H,) 



th ,i^. *-A: f°« '-rners, w. can obtain 

■ DOT o/oTd m't'" new ■i,l.=fc B. -directly ^rU'tne 

20, domain. translatidn equation Is' dsfinedV 
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Matrices H.^ and have similar structures to those of 
and described earlier* ' A sub-matr^ix at the lower-left 
corner or the upper-right corner forms an identity matrix. 
The DCT of H.-^ and may. be pre-computed and stored in 

5 memory, since there are a 'finite number of values. 

The computational complexity of the DCT domain 
translation equation is of ten reduced in view of the. fact 
, that many ^elements of DCT,(Bj) '. are- 'typically zeros Since 
... most DCT coefficients. .are . further encoded with the run- 
10 r ;l,^ngth code, the locations of non-zero DCT ' coefficients 
, . can. be easily found _by, utilizing their- ^run-length formats 
If the motion vectors are zero or: integral^ multiples of 
..the. block width,., t^^^' block structure alignment procedure 
is, not , necessary Motion compensation in the DCT domain 
15 , requires simple additions: only,- -as; in the^ spatial doiriain. 
If -one motion : .vector J ?component. (d^' or d ) -is' zero' or 
. integral , multiples iof r -the ' ' ^ block/ width ^ -- the DCT 
. coefficients, of the new blockjican 'be computed from the DCT^ 
coefficients of only two original blocks, rather' than four 
20 blocks. This results in the simplified transform domain 
equations: , , 

DCT{new block) =DCT (H^^ ) DCT ( ) +DCT (Hg^ ) DCT (Bg) if d^=0 
DCT(new block)=DCT(B^)DCT(H^2)+DCT(B2)DCT(H22) if dy=0. 

Thus, the operation of . the. .transform , , domain 
translation equation, and its simplified anal.og equations, 
are required within the reference block. calculator 61 • of 
decoder 60. Accordingfly , , reference block calculator: , 61 

30 may be viewed as including a transform domain translator 
67 for executing the transform domain translation equation 
and/or the , . simplified, transform v domain translation 
equations. Specifically, the transform domain translation 
equation is used to obtain the DCT of the reference block, 

35 i.e., DCT(P^^^(t-l,x-d^,y-dy)) . 

As shown in Figure 9 , the outputs of the transform 
domain MC decoders 60 are conveyed to a transform domain 

^■15 - 
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■ compositing unit" 62V in kccdrdance ' with^'tifte ' iVi.>ention, 
the transform domain compositing unit =62 "provides number 
' . i of novel compositing . operations Whicli aire perform on 

■■• t-ransform domain vdded signals. ■ As' • will ' be' fuii^ther 
. 5 described below/. the trartef drm domain ^doiripbsiting' unit 62 
> ■ ...performs compositing operations directly In the transform 

V domain; . Thus,: the .computational ■ overhead associWte'd with 
1 - transformation'tothfe spatial domain is nbt te^uired. m 

" a<3<JitlonV since ■comp5^iting - 6i>erations== are perfirme^ on a 
10 compressed data- signal, the' amount of dWta processing is 
■■.reduced. -^^ • ■ ■ .i - .. 

■ • After, eompositing in the'-' bCT doma'in, as "vill be 

described below, the compos ited^ Signal is conveyed to a 
. . transform domain MC ■ encdder ^ U '/ ' ' hig^r^ ' " depicts a 
15.., transform dbmain-'MC ^enWet ^4- -'in acborkance with the 
. invention . Th4' tirankf binA- dbii^in MC •^ncbaer 64 caiculates 
the prediction error' 'and the ^ inotibri-' Vector' signal This 
- - operation can be mathematib'atiir ^chajracterized" by the 
.! ■ transforin domain MC encbdeV- equation ^ ' ' ■ 



25 



30 



35 



•The, variable d...., is^ the new motion vector for the 
composited video: secjuence. - As 'shown "in Figure 12 the 
motxon compensator 68 identifies the new motion vector for 
othe composited video sequence. ^ ' This 'mbtiori vector 
Identifies the reference block Tn the pr^viou^' frame/ and 
■the reference, block , is' subtracted ' ftoxt,- the input 
composited signal to yield the tebr' signal outpvit for the 
encoder (DCT(e(t,^, y) ) . Note that' k quknti^er (not shc^wn) 
may. be , used to processor the error ' 6uti)ut. ' 

The .frame^ memory 69^' stores a" previous frame value 
accounting for the (t-l) Valub: Nbte, all ' t^rms in 'the 
above, equation. , are in thb' 'DCT-dbmaln. Since tKe new 
motion vector may i-not 'bb 'integral' multiples of the block 
wxdth, . the transform domain tra^nslation equation may need 
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to be executed, in order to calculate ; OCT coefficients of 
* the refe^rence . block (DCT ( P. { t-1 , x-d,V^> y-d ' yO ) ) • • 

^ New motion vectors d J. > of . the- composited images can be 

, obtained by inference from^ the .original (motion vectors. 

.5 This, proqess.^-is. .illustrated: -in reference, to Figuire 3, 
which c^epipts , the problem; t of - a background image being 
replaced by a foreground, .image •j. . D^j is . the location of the 
current block and D ,is , the location of the original 
optimal reference location, r^he . new location : of the 

10 reference block can, be either^ D,., or. D,. As shown in 
Figure 3, is the location of the first background iblock 
in the overlapped , image horizontally > displaced- from the 
original optimal reference location D, and . ,D , is the 
location of the , first, background block in, the overlapped 

15 image vertically .displaqeid . . f rom the ^original . opt'iroal 
reference location R. , Tlie,. choice pf the. reference block 
is based on the assumpjti^on. that .jthe distortion, b the 
current block (DJ .and | ppssibl§^.re;f erei;ice= blocks, increases, 
monotonically with, the .^if feipence, in location between' the. 

20 current block and the possible reference blocks. In oth^r 
words, the new ref^?:,e^iqe block .se-l^cted will^ ber that block 
(D^ or D^) whose image content is closest to the current 
block (D^) . In , ;the.; slzriplest i approa-ch, we ban use 'the 
current block. 1 oqa.t ion .(D^) as^. the new reference block, r 

25 Returning ..nQW.l t,o figure . 9 , ,the . .er*ror* and 'motion 

vector output^ Sj^igna Is of .the. transform domain »WG encoder 
64 may., be conv^y^d 9yer. : ,,transmssion^ cha?nnel » 28E to a 
standard MC^DCT ,dec9de^.j,52B. ; The output of the MC/DCT 
decoder , 52B is;, a recp^n^tr,uct^d , video signal which >may be 

30 displayed on yidep dippl^ jd^yice,56B. j > ' ' 

Thus, a priin,ary ,. aspect of .the; present invention has 
been described. ,Nainely,. the > operation of transfoirm domain 
MC decoding^ and ^ encoding . of a transform - domain motion 
compensated signal, has been^ described.! IChe ~* trans form 

35 ^ domain MC decpdjLjag anji, . encoding \ u operations* ^ * allow' ' 
processing^ of video , , signals :in, the- tiransform domain. 
Another aspect of the invention is to provide a 
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methodology for ^actually ^.erf o^ing cdmpi^-iting^ operations 
in the transform domain. Thus, attention' presently turns 
to this aspect of the invention; -\ • , ^ • 

The transform. domain . compositing' unit ' of the 
i 5 , invention: supports: a number; of compositing^ operations . in 
particular, the compositing unit 62 incMdfes a transform 
. domain translator 67 /"a. transform domain bverlap ^x^cution 
unit 8o; a transform -domain pixel-wise multiplier 82, a 
transform domain filter unit 84, and' a transform dbmlin 
10 scaling unit 86, These units are depicted iri Figure lis', 
■ which. will be more fully described belowl Note th^t in 
Figure. 13 "TD" is short for "t^ansrorm domain"! - - 

The important compositing technique' of translation 
can be achieved by using the previously described 
15 transform domain translation! elation. ' In' general, if a 
video object is moved a distance of an arbitrary number of 
pixels, the: DCT-coefficiehte^6f the' video 'objects will 
. . become " non-aligned,' with respect to the block structure 
> • used for the .final display video signal . A new block of 
20 translated video signal; generally has contributions 

,• from four., original old. neighboring blocks, such as B,; B^, 
B3,...and.B, in Figure. 2B. ^ iri ordet to 'align ' the' Dci 
coefficients to the final , block' structure/ the previously 
described transform domain, translation equation ma^ be 
25 used to calculate the DCT coefficients of 'the new video 
block, B'. That is, the transform domain translator 67 is 
used to execute the transfprm domain translation equation 
However, in some simpler applications, translation 
, distance may: be, restricted to ^integral multiples of the 
30. block : width. in • that case,, the transform > domain 
translation equation operation is not nbcessary. Moving 
a.video object:,simply. requires changing the criginal point 
-position of the video ^ object. ^ \ , . 

Another transform domain compositing fiinctibn is 
35 overlapping. m the present invention, overlapping^ 
operations are preformed by a transform domain overlap 
execution unit 80. Opaque overlapping of ' two video 
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. objects requires substituting pixels of the foreground 
video . object for ..those . of, the j background object. , Semi- 
transparent overlapping.! requires a linear combination of 
the foreground/ and .background pixels-, i^e;, , P^^^ - aP^ + (1 
5 - cc)P^r, ^f^p.^l^^O.^J\^y, . '^s' and; are. new pixels/ foreground 
pixels,, and background: pixels^ .respectively;.w ^.The term a 
represents the. percent.age .of opaqueness desired. Thus, an 
a value -of 100% or^ (1. 0)..;^, ;T>^ould . r.epresent. an. opaque, 
overlapping operation.,: ; • . . 

10 . Since overlapping is a linear operation, the. spatial 

domain technique ^ may- be used in i the ■■ DCT- domain, 
specifically, ,a DCT. ^.dpmainj * overlapping . equation may be 
-defined as follpv/.s: . / ,p _ ;. . 

15 . . .r PPfr(P„^^)j r^,,QtDCi;(P3) .(1-05) DOT (P^^) ^ 

.If, the ;a cqef f ifientp,. .yary:i f rom pixel- to pixel, it 
, beppmes a pixeljwise operation . v.. An equivalent operation 
is pixel-multipliGatipn, , vVhijch v is = performed by -^ the 
20 transform doitiain i:p.ixel-wise;^^. m^ :: 82 - of the 

invention. Other, feaitures(; requiring pixel-multiplication 
. , include s\^btitles,. .canfci-aliasing. for the boundaries of 
irregularrshaped ^ yivdeo.c .objects, i and ispeciai-ef f ect 
masking.. . . ,The ti^ansfpjnri . domain .pixe^^^^ multiplier 8,2 

25 . .executes th^ following . equation: = . - » j ^ c . 

, To. computj^, J ipi-x^l-^wise ^multiplication • in the ' DCT 

3,0 . domain^ a multipl^icatiou-convolution relationship f or^ the 
DCT is deriyed^ * . ^ M - • 'i-^' ^ • - — *^ 

. J Suppose.^X^ is ,thej.DGT of .image , block X.- -An extended 

symmetrical version of the DCT^ coefficients may be formed 
follows:.; .\ , •> . ;^ . ,v - . ^. 



{ ,,k2 ) /c Ck, , kg.) . . ., k, , kg. - 0 , . . , N-- 1 
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■ ' • (■ -^k- , kj ) / C (-k, , k-) '■ • k,=-N+l ; ; i . ■ ^ 1 ; k2=0 , . . . , N- 1 

X^ (k, ,-^k2)/c'(k,/^k2) • ' -'^^^ k,=0, . ; /.N-l'; k2=-NVi, . . i , -1 
• ■ ' ■ X^"(-k,V-k^) /C (-k,^ -k2) ' ^ ^ • ' , k2--W+i , -1-^ ■ - . ■ 

where C(k) = s,. .. , ^ 



when k=0 and 1 when k=l,, ...N-1. 

. ^ .Then, the multiplication-convolution t^^ be 
10 described as follows: 

If y(m,n)=x(.m,n)*hj;.m,n)^ for^^ then , 



( (k^-n) ) ] *a (k^-m) a (k,-n) 



15 



Where kl, k2 = -N,...,N-1, and .a(i) = i when i e [-n,n-1], 
otherwise a(i)=-i. Note that the syntax ((n))„ represents 
^ the operation of n modulo. N. 

If the a variables are ignored, the aboye equation is 
20 equivalent to a 2D 2N-point circula?: convolution. , In 
other words, the NxN DCT coefficients of an image block 
are expanded to a syipmetric 2N x 2N extended block. The 
pixel-wise multiplication . . of two image c blocks in the 
spatial domain corresponds to the 2-diinensional 2N-point 
25 cijrcular convolution-like operation,. of , the extended blocks 

in,,the,.DGT domain,. ■ . ,, . , > - . f ,.. 

Another .oqinpositing technique wh-ich. can be performed 
in the DCT domain is two-dimensional separable linear 
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10 



15 



filtering- This operation is performed by transform 
domain filter 84, 

For simplicity, a one dimensional .filter . is initially 
described, , If ^,x[n] represents ; the one dimensional image 
sequence and .h[nj. represents the. filter coefficient 
vector, therlt the output filtered image sequence, y [n] , can 
be calculated as the convolution of h[n] and x[n]. 
Namely, y [n]=h[n]®x[n] . . . . 

Figure 14 assists in illustrating this process. Each 
element of the output vector, y[n], is the inner product 
of the filter coefficient vector and a corresponding 
segment of x[n] . If the image sequence is segmented into 
N-element segments, for example x*^ representing the kth N- 
e'lement ■ segment in the original image sequence, 'then the 
above equation can be described as' the one-dimensional 
matrix-vector multiplication equatiion: 



■; SZ~1. 



where 





J." 




' 1^ 




, S2= 













20* 



25 



and H-* are NxN matrices, as shown in Figure 14 1 If all 



matrices are coricatehated side' by side to form a new 
matrix, H, then matrxx H can be defined as: 

H(l;3)=h(j-^i^N-rK' if -1^^ 

.■1. i=0 -v-^.-M '-vw, otherwise' ' t. v-. 



. : Taking 'the transform operation on both^ sides of the above 
equation and inserting an inverse transform/ transform pair 
3d front of : the - input segments ,^ we eanf- obtain : ^ 
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and are NxN matrices, as shown in Figure l4 . if all 
matrices are concatenated side by side to form a new 
matrix, H, then matrix H can be defined as: 

5.-. - H(i,j.)-h(a-sl*Nnil,,;;if,nli<=j^-sl*N-i^^ , 
~0 ^ otherwise , . 



Taking the transform operation on both sides of the above 
equation and ii;iserting an inverse transform/transform pair 
lO in front of the input segments, we' can obtain; 



S2-1 



If we use pi to represent the matri.x'.prdduLt T^HiT/V (which 
15 can be calculated in advance) X to i^epresent T^x, and! Y- to 
represent T,y , then the above . equation Jpecomes . 



■ J'-Bl 



,1 ! i .V 



20 Note that and .T^ can be. the. same transforms such as the' 



OCT.. 



, , To,. extend to, a two-dimensional separable filtering, 
the matrix-vector multiplication' is changed to matrix- 
matrix multiplication with pre-matrices aiid post-matrices. 
25 The. one dimensional .matrix-vector multiplication eijuation 
becomes, the two dimensional matrix-matrix multiplication 

_ .e.qu,ation: .>.,,■ ■ . .... , . ; . , 
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where H represents vertical filtering (by column) and W 
represents: horizbntal filtering (by tow)'/' The 'equivalent 
operation in the transform domain is as follows: 

S2-1 t2-l 



where T- can be any transform matrix. For example, in the 
DCT algorithm, T^'\ is the DCT transform matrix and 

T^=T2^. Thus, the transform-domain two dimensional 

10 filtering can be calculated as a summation of several 
matrix' products . 'Ag^in^ '%he' pre-matrices T^H^T^'^ and post- 
matrices T^*''w^T2 calculated in advance. 

Anothdt ifnpdrtarit' " image manipulation technique is 
scaling. Scaling in the transform domain may be performed 

15 by a transform domain scal-ing unit. 

Theoretically, scaling can be completed by linear 
filtering plus resampling. Each pixel in the final scaled 
image is a linear combination of all pixels in the 
original image. A simplified way to implement scaling is 

20. -to. use a box :area' averaging technic^ue . ' i Specif ically, 
scaling with a rational factor, say p/q, simply maps q 
original pixel ..areas txDi p ; target- g>ixer -area p and 

q, are ,two . integers, without common f actors greater than 
pne> , , Therefore , ... each* target pixel* ca:rr- be - expressed as a 

25,- lihe.ar cpmbin&tiori ^of a ' limited number' "of ^original 
neighboring pixels*, with the' sum of ^ wei^ghting coefficients 
equal to one. For the one dimensional scaling Operation, 
the relationship between the original pixels, x, and the 
new pixels, y, can be characterized by the one-dimensional 

3 0 matrix-vector multiplication equation, described earlier* 
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blocks, are the vertical scaling -jnatriceS, ^nd W.' are 
the horizontal- scaling, matrices. Fot example: : 



^1 = 



0.50.5 0 ■ 0 

' 0"-'' 0 ■ 0.5 0.5 
0 ^^0 0 0 



I ■ 



0 ^0 
0- ;.0 
0.5 0 . 5 



0 ^ 0 
C 0 
0 0 



.. P ,, p,,5. 0 .5. 



. 1 



10' 



15 



20 



25* 



This equation is applicable 'for 'a -block' width of V'pixels 
Sxnce these operations' a¥e ■■liri4ar; thei^ x^ay b4 performed 
in'the 'DCT dbmaih. ' ' ■ ^r. - j 

y in ■tri^ art^vair aiso recognize that the 

^dxsclosed compositing techniques can be applied' in general 
■transform domairisv. like the blPT^ ti^ansfSrm and the DST 
tran^fdrm. m addlfion/ the^dlsBlosei techniques may be 
used .for • ^rnage' format ^coHv^tk ion' Wtv^een different 
compression stahdi-rd^, ^uch; as feWtW^en MI^EG and jpeg ' 

one skilled-in thd art will recognise that the 
transform domain compositing ' operat^iohs, ' the transform 
domain decoding -and encoding ^opex^ations, and the other 
■operations associated with '•'the invention may be 
implemented by usihg standar^d ' 'hardware devices and 
prbgramming techniques. Fi^uz^g 13 'depic:ts an apparatus^ 
which may b^ used to execute'' the functions of the 
invention. The appa^atis inci^de^ " a i^ntral proc^^sing 
unit 70 whldh is coupled td a memory ' module 72 ' The 
memory module 72 may' b'e any dombin^tion of ' 1^/ rom^^ and 
dxsc memory devices . The memor^' mc^t;:iW 72 stores ' a niamber 
Of programs for fexecuting the operations 6^ the ihventkbn. 
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Specifically, J tfee xnemory module 72 stores the execution 
units of , the transform domain : MG. decoder 60: the inverse 
quantizer 34D, reference block calculator 61, and frame 
memory 63. The memory .module 72 stores the execution 
5 units of the transform ^domain MC encoder 64: motion 
compensator 68 and frame memory 69- The memory module 7 2 
also stores thei execution units of the transform domain 
compositing unit 62: transform domain translator 67, 
transform domain overlap execution unit 80, transform 
10 domain pixel-wise multiplier 82, transform domain filter 
84, and transfor^n domain scaling unit 86. 

i 

The apparatus of Figure 13 also includes known 
interface devices 74 to provide an interface between the 
digital processing elements (CPU 70, memory 72) and the 
15 video processing elements, such as transmission channel 28 
and video display crevice .56B., . . ; , 

Now that the invention is fully .disclosed, attjention 
turns to peripheral issues surrounding^ implementation of^ 
the invention. ^ The - . most , complex :Operat;ions= in the; 
2 0 transform domain are the transform domain MG,..- encoding 
transfprm domain MC de.cpding, ai^d pixelrrwise^, -translation 
operations. The com^jlexity, primarily results from 
computing the block structure adjustment proce.duire. In 
the case of the. trans form , domain MC encoding and decoding, 

2 5 if the non-zero motion vector;, percentages; aire, decreased, 

computations can t>e . reduced . . . , ^ 

In the regular ■ MC al^gorithm;, the^ ^on.-2;,ero_ motion 
vector percentages vary wit:h dif ferent video. - .sequences . 
One way to reduce the non-zero motion vector jperoentages 

3 0 is to give some preference at tl^e transmitter ,,to the 

reference block with zero motion vectors An .the MC 
algorithm. For exampjl^,, , instead of. Cprcing, the^ motion 
vector calculator 53 to search for the opt iit|al. reference 
block with^the minimal .block distortion, we c^n.,ad4 a rule 
35 forcing the ^qptimal reference block^ , to, have a block 
distortion value, smaller than 80% of . that for |the zero- 
motion block (hereinafter the 8 0% MC approach) . 

^: 25 - 
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Oth'erwise, the ' zerb-motion block ' is ^ selected "as the 
reference block. ' ' ' ' ' ' ' ; ■• 

• Another- possible 'technique ^ 'tb'' reduce' the 
• • computational^ comprexity ' in the D'CT d'orria'in i'd to combine 
5 a sequence -.of. operdtiorts- -viri^ip • a . singlte '"o^ratToTi:: "'For 
• ' ®^*»Pl-®> ' MC/DCT 'comt)res^se'Q ' images, ■ We Way heed to 

perforTnv F{DCT'(e) + G(DCT(p- ))•) , ' where G represents the 
reference block calculation in 64, and F represents the 
scaling operation, since they are both linear operations, 
10 we can apply the distributive law to change the above 
formula to F(DCT(e) ) + H (DCT (P^^,) ) , where H represents the 
composite function F*G. The computations for function F 
can be reduced since the non-zero coefficient percentage 
for DCT(e) is usually higher than that for the 
15 reconstructed image, e.g. DCTfP ) 

^ ref ' * 

The disclosed DCT-domain compositing approaches can 
reduce the required computations for DCT-compressed images 
by 60 to 75%. The required computations for MC-DCT 
compressed images may be reduced by lo to 25% The 
20 results depend on the compression ratios and the non-zero 
motion vector percentages. That is, the compositing 
throughput is variable from image to image. However if 
constant-throughput video compositing is desired, one'can 
skip the high-order DCT coefficients whenever the 
25 processing delay bound is exceeded. For example in 
compositing mc/dct compressed video, if the input video 
sequence has a very high percentage of non-zero motion 
vectors, one can simply process important low-order DCT 
coefficients in each DCT block and skip high-order DCT 
30 coefficients. By keeping low-order DCT coefficients, one 
can minimize the image quality loss. 

The foregoing descriptions of specific embodiments of 
the present invention are presented for purposes of 
Illustration and description. They are not intended to be 
exhaustive or to limit the invention to the precise forms 
disclosed, obviously many modifications and variations are 
possible in view of the above teachings. The embodiments 



35 
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were chosen and described in . order to best . explain the 
principles of the invention and its practical applica- 
tions, . to thereby enable ot^hers skilled in .the art to best 
utilize the. i^n,vention and, various embodiments with various 
5 modifications ^as are suit^ed ,,tp the particular use con^ 
templated. .^It is intended that the scope of the, invention 
be defined by the followingiClaiins and their equivalents. 
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IN THE CLAIMS: • ^ ■ ; - 

1. An- apparatus' for- : process irtg " a • f irst * motion 
compensated transform domain video signal and a ^ second 
1 motion compensated transfoinri domain' video ' bignal , said 
5 .apparatus comprising: • : . >r. ; r, ;.) ; n. : ■■ 

an inverse, motion compensating ' device ' operating on 
-said first mot ion cdinpenSated ' trains form ddihai'n video 
signal and said second motion compesnsated trahsf orm domain 
video signal such that • said^' first motion compensated 
transform domain video' signal is" transformed to' a first 
transform domain Video sighalV " and said second motion 
compensated ttarisform ' domain video signal is' transformed 
to a 'second trahsform'domairi 'video signal, said inverse 
mot idn ^compensating device ihcl^tidi'ng ' transform domain 
15 - object translator for Garcula'ting hon-girid ' aligned 
reference -blocks within-a • grid • of "a • f irst video 'framte ' of 
-said first transform dorfiaiin video signal 'to irehder a 'grid 
, aligned current block in *a grid of a seeond" video' frame of 
said first transform domain video signal; and 
20 ta compositing mecKanism cbuifjled- to said inverse' 

motion compensating , device said compositing mechanism 
performing transform" domain Vldeb dompositing operations 
on. said first transform domain video" signal and on said 
second transform domain video signal' to render a transform 
25 domain composited video signal " ? > 

2. - The apparatus of . claim '1 wherein ■ said tr-'ansform 
domain object translator generates DCT coefficiehts for 

V said non-grid aligned reference' blocks b^sed 'upon DCT 
30 coefficients of corresponding adjacent grid-aligned blocks 
encompassing each- of said non-grid 'alignfed reference 
blocks. ... • (■ >■ ■■ . I 

3. A method of -processing' a first video "signal and a 
35 , second video signal, .to /form a single composited video 

! signal said method: comprising, the'v steps of: ■ ; 
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encoding said first video signal at a. first location 
to form a first motion -compensated transform domain video 
' . . signal ; • . - : • .1^ ; . ■ . »; <• • 

.encoding,.^ isaid second • -^video .signal., .at a. second 
5 location to form a second motion; ..compensated transform 
domain video signal; , . . v - 5 i : : 

transmitting said f irst.imotipn compensated; transform 
domain -v.ideo^ signal to, a third location;. 1^ ( 

transmitting said second motion compensated transform 
10; domain video signal to, .said third localiion; . , 

executing an inverse motion compenisating operation at 
said third location,^ such. . Jihat, said , first, motion 
compensated transform,, .d9main videio,, signal is , transformed 
to, a first trans form, d9jna in video .signal ^ a-n.d said se^cond 
15. mot4.on compensated pompi:es;sed video signal is .transformed 
, to a second trans fbirm dprnai^^ video, i signal , = . isaid executing 
sj:ep inclufaing . the ; st,epi:,.Af . tr non-grid aligned 

reference jDlocks within a grid; of ,a first video frame o# 
said first transfprm domain video signal -to render a grid 
20 , aligned current block in a. grid- of,, a /second video frame of^ 
said, first transform domja in video, signal and ^: . 1 h < 

^ performing - transform , dpinain ;video^ compositing 
operations on .said first .Jiransform domain video signal- and 
on said seppnd transform,^ domain video signal to render a 
25 transform domain composited video signal.^ ^ . :• 

4,, The method qf, claim 3 further ^ comprising: the. step of: 
. . encoding said atr^msform .domain l x^omposi ted video 
signal^ to, :^orm a mot-ipn. epmpens domain video 

30 :sign,al ; ._and., . : . Vr-. ^ • - 

. transmitting said motion compensated transform domain 
video signal to a fourth location. • > - ' 

5. . : The, .mej;hqdf,pf cl^im 4 . further comprising v the step of 
35 .. . . : ef fecting , an ^ inverse transform domain loperation and- 
an inverse motion .compensation .operation, on- said motion 
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